[FEAT] Fix projection pushdowns in actor pool project #2680
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jaychia
force-pushed
the
jay/app-project-pushdowns
branch
from
7378183
to
12fcb9b
Compare
jaychia
force-pushed
the
jay/stateful-udf-logical-opt
branch
from
772c9a2
to
19dca2d
Compare
jaychia
force-pushed
the
jay/app-project-pushdowns
branch
2 times, most recently
from
cfeb1bc
to
2040828
Compare
kevinzwang
approved these changes
Aug 23, 2024
Comment on lines
247
to
254
| let required_column_names = projection | ||
| .projection | ||
| .iter() | ||
| .flat_map(get_required_columns) | ||
| .collect_vec(); | ||
| let distinct_required_column_names = | ||
| required_column_names.iter().collect::<IndexSet<_>>().len(); | ||
| if required_column_names.len() == distinct_required_column_names { |
There was a problem hiding this comment.
Very minor comment but a possible way to do the same check that may be more performant is to iterate through the required columns and put them into a set after checking if they already exist in there. That way you can early terminate
| .arced(); | ||
| let plan = LogicalPlan::Project(Project::try_new( | ||
| plan, | ||
| vec![col("udf_results_0"), col("udf_results_1")], |
There was a problem hiding this comment.
You could also add some aliased columns in here too, since they do not perform computation so you expect them to also be merged with the ActorPoolProject correctly
There was a problem hiding this comment.
Good call-out, added it into the test and verified that the alias gets correctly absorbed into the ActorPoolProject
jaychia
force-pushed
the
jay/app-project-pushdowns
branch
from
2040828
to
b76b148
Compare
CodSpeed Performance ReportMerging #2680 will not alter performanceComparing Summary
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds some fixes for performing projection pushdowns performed by actor pool project